
For training and testing our SRL systems we used a version of the CoNLL 2008 shared task~\citep{surdeanu08conll} dataset that only mentions verbal predicates, disregarding the nominal predicates available in the original corpus.\footnote{The reason for this choice where license problems.} While the original (open track) corpus came with MALT~\citep{nivre2007mli} dependencies, we observed slightly better results when using the dependency parses generated with a Charniak parser~\citep{charniak00amaximum}. Hence we used the latter for all our experiments.

%we modified the PropBank corpus of the CoNLL-05 shared task 
%\citep{carreras05introduction} to be compatible with the format of CoNL-08 
%shared task \citep{surdeanu08conll} which uses dependencies trees rather than 
%constituents. 
%IV+
%We choose to work with dependencies since recent development of dependency parsers had made possible for dependency-based SRL systems
%to reach state of the art performance \citep{johansson08dependency}. 
%We use the Charniak parses available in the original corpus and 
%converted into dependencies using the software developed by 
%\citet{johansson07conversion}. 

To assess the performance of our model, and it to evaluate the possible gains to be made from considering a joint model of the complete SRL pipeline, we set up several systems. The \emph{full} system uses a Markov Logic Network with all local and global formulae described in section \ref{sec:model}. For the \emph{bottom-up} system we removed the structural top-down constraints from the complete model---previous work~\citet{riedel08collective} has shown that this can lead to improved performance. The \emph{bottom-up (-arg)} system is equivalent to the bottom-up system, but it does not include any formulae that mention the hidden $isArgument$ predicate.

For the systems presented so far we perform joint inference and learning. The \emph{pipeline} system differs in this regard. For this system we train a separate model for each stage in the pipeline of figure \ref{fig:achitecture}. 
The predicate identification stage identifies the predicates (using all local $isPredicate$ formulae) of a sentence. The next stage predicts arguments and their roles for the identified predicates. Here we include all local and global formulae that involve only the predicates of this stage. In the last stage we predict the sense of each identified predicate using all formulae that involve the $sense$, without the structural constraints that connect the $sense$ predicate to the previous stages of the pipeline (these constraints are enforced by architecture).    

%This model simulates a  
%pipeline system, in which decisions for ML predicates of the first statges have
%stronger effect on the predicates of latter stages. However, this still is
%joint model. 

%Our third system, \emph{bottom-up (-isArg)}, departs from the second but it ommits the \emph{isArgument} hidden predicate and the formulae associated to it. With this modification this model follows a traditional approach where SRL system do not model the plausibility of a token of being argument.
% IV: Note, the results section says: this aspect of our model requires us to
% jointly perform inference for all predicates of a sentence. Dunno how to
% explain that.

%Finally, the four system, \emph{pipeline}, is a pipeline system. In this case, we
%create three models for each stage of our architecture (see Figure \ref{fig:achitecture}), and we pass the output of one as input of the next stage.
